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ABSTRACT 


The  paper  points  out  that  in  a  decision  task  involving  the  selec¬ 
tion  cf  a  system  from  a  number  of  competing  systems,  a  single  number 
or  figure  of  merit  is  usually  inadequate  to  describe  the  merits  of  a 
given  system  and  that  a  matrix  of  numbers  representing  the  important 
parameters  is  required.  A  description  is  given  of  the  parameters  in 
this  matrix.  Rules  are  given  ior  reducing  the  matrix  and  for  drawing 
conclusions  from  the  reduced  matrix.  An  example  involving  a  develop¬ 
mental  decision  is  included  to  illustrate  the  process. 

— \ 

Any  views  expressed  in  this  paper  are  those  of  the  author.  They 

should  not  be  interpreted  as  reflecting  the  views  of  The  RAND  Corpora¬ 
tion  or  the  official  opinion  policy  of  any  of  its  governmental  or  pri¬ 
vate  research  sponsors.  Papers  are  reproduced  by  The  RAND  Corporation 
as  a  courtesy  to  members  of  its  staff. 
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INTRODUCTION 


While  the  general  method  of  decision-making  presented  herein  has 
broader  application  than  just  to  the  developmental  decision  problem, 
it  is  discussed  in  connection  with  this  problem  to  enhance  clarity 
through  dealing  with  the  concrete  rather  than  the  abstract. 

Systems  are  generally  evaluated  on  the  basis  of  their  ability  to 
perform  some  prescribed  mission  or  missions.  The  representation  of 
each  system  by  a  single  number,  like  a  figure  of  merit  or  a  cost- 
effectiveness  value  would  simplify  the  selection  process.  However, 
as  pointed  out  in  Ref.  1,  this  is  usually  inadequate  to  the  task,  and 
the  decision  maker  should  be  supplied  with  a  group  of  numbers  for  each 
system  being  compared.  This  group  of  numbers  will  be  called  the  deci¬ 
sion  matrix.  If  one  lists  in  the  decision  matrix  all  the  factors  that 
have  a  bearing  on  the  selection,  the  matrix  often  becomes  excessively 
unwieldy  and  confusing.  On  the  other  hand,  too  brief  a  list  might 
eliminate  an  item  essential  to  a  proper  decision.  The  purpose  of  this 
discussion  is  (a)  to  indicate  the  parameters  that  should  be  included 
in  the  decision  matrix,  (b)  to  discuss  the  procedure  for  reducing  the 
matrix,  and  (c)  to  provide  rules  for  drawing  conclusions  from  the  re¬ 
duced  matrix. 

The  rules  for  the  matrix  reduction  and  the  judgment  processes  are 
considered  to  represent  only  an  initial  effort.  Amplification  and  re¬ 
finement  of  the  rules  can  be  expected  as  experience  with  these  processes 
grows,  involving  a  wider  variety  of  cases  and  the  contributions  of 
people  with  greater  insight. 

The  valuable  comments  of  Mr.  Kenneth  R.  MacCrimmon  and  Dr.  Olaf 
Helmer  are  gratefully  acknowledged. 
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GHOICE  OF  PARAMETERS  FOR  DECISION  MATRIX 


The  decision  matrix  should  be  complete  in  the  sense  that  all  in¬ 
formation  having  an  important  bearing  on  the  choice  should  be  included. 
The  number  of  parameters  in  the  decision  matrix  can  be  kept  in  hand  by 
combining  under  a  single  generic  parameter  a  number  of  related  para¬ 
meters  representing  sub-items  under  the  generic  parameter. 

This  combining  of  a  number  of  related  parameters  into  a  single 
parameter  can  be  done  by  employing  a  figure  of  merit.  A  brief  discus¬ 
sion  of  figures  of  merit  is  given  in  the  Appendix.  It  is  important  in 
selecting  the  subparameters  that  are  associated  with  any  given  generic 
parameter  that  there  exists  a  true  relationship;  in  order  words,  one 
should  try  to  avoid  mixing  apples  and  oranges  under  a  single  figure 
of  merit,  but  should  limit  the  process  to  various  kinds  of  oranges. 

The  following  are  the  parameters  I  believe  appropriate  to  a  decision 
on  system  development. 


Decision  Parameters 


Mission  initial  operational  date  (IOC):  The  date  on  which  a  mis¬ 
sion  operational  capability  is  required  is  a  major  factor  in  setting 
the  urgency  for  the  decision  on  system  selection  and  the  state  of 
art  chosen. 

Decision  date:  This  is  obtained  by  subtracting  the  estimated  time 
required  for  R&D  and  total  system  construction  and  assembly  from 
the  mission  IOC  date.  It  provides  an  estimate  of  the  time  available 
for  research  and  exploratory  engineering  to  improve  the  state  of 
technology  or  the  confidence  in  a  new  technology  before  a  decision 
is  required. 

System  R&D  cost:  This  is  a  measure  of  the  investment  risk  in  a  new 
system. 

Mission  cost  effectiveness:  This  is  a  major  factor  in  determining 
the  worth  of  a  new  system.  It  can  be  expressed  in  two  ways,  de¬ 
pending  on  the  situation  appropriate  to  the  problem. 

(a)  The  cost  to  accomplish  a  prescribed  objective; 

(e.g.,  in  a  study  of  the  relative  worth  of  com¬ 
peting  aircraft  propulsion  systems,  the  cost 
effectiveness  may  be  measured  by  the  cost  to 
transport  a  given  amount  of  payload  a  given 
distance  over  a  stated  number  of  years) . 

(b)  A  measure  of  mission  accomplishment  for  a  given 
total  investment  in  the  mission;  (e.g.,  the  size 
of  space  station  that  can  be  built  on  the  moon  in 
terms  of  weight  of  equipment  delivered  for  a  given 
budget  allotment). 

If  the  system  under  analysis  is  part  of  a  more  complex  system  re¬ 
quired  for  the  mission  (e.g.,  a  propulsion  system  in  an  airplane) 
then  the  mission  costs  should  include  the  R&D  and  procurement  costs 
of  the  system  under  analysis  and  also  the  mission  operation  costs 
and  any  R&D  and  procurement  costs  associated  with  the  modification 
or  procurement  of  the  total  system  that  may  be  pertinent  to  the  case 
under  consideration. 
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The  effect  of  system  reliability  on  mission  cost  may  be  handled 
in  two  ways. 

(a)  When  mission  costs  are  extremely  high  or  mission 
failure  not  permissible  because  of  human  cargo, 
differences  in  basic  reliability  of  the  several 
systems  under  comparison  are  taken  into  account  in 
the  cost  by  introducing  sufficient  redundancy  in  each 
case  to  achieve  a  specified  probability  of  success. 

(b)  At  the  other  extreme  where  mission  costs  are 
moderate  and  mission  failure  tolerable,  repetition 
of  the  mission  rather  than  the  introduction  of  re¬ 
dundancy  may  be  the  most  practical  and  least  costly 
approach.  The  mission  cost  for  this  case  is  obtained 
by  dividing  the  mission  cost  for  100  percent  system 
reliability  by  the  estimated  system  reliability. 

Whether  one  uses  procedure  (a)  or  (b)  is  an  individual  consider¬ 
ation  for  each  case.  And,  in  some  cases,  a  mixture  of  these  two 
approaches  may  be  the  least  costly;  i.e. ,  it  may  be  advantageous 
to  introduce  some  redundance,  ;even  if  repetition- of  the  mission 
is  permissible. 

o  Confidence  of  meeting  mission  IOC  date:  The  confidence  of 
meeting  the  IOC  date  can  be  estimated  from  a  consideration  of 
the  number  of  components  in  the  system  that  are  required  to 
attain 'advanced  performance.  If  all  components  are  required 
only  to  attain  current  performance  or  modest  extensions  thereof 
then  the  confidence  is  close  to  100  percent.  When  the  require¬ 
ments  on  some  components  are  advanced  w  11  1 «. youd  current  state- 
of-art  and  also  when  novel  components  with  no  prior  developmental 
background  are  introduced,  then  confidence  in  meeting  the  IOC 
date  decreases.  For  example,  at  the  start  of  the  development 
of  the  aircraft  nuclear  propulsion  system,  a  low  confidence 
estimate  would  be  appropriate  because  of  the  novel  high  tempera- 
-ture  nuclear  reactor^  The  confidence  is  expressed  by  a  number 
between  1  and  0  where  1  represents  perfect  confidence. 


o  Flexibility:  While  the  primary  mission  task  may  be  closely 
prescribed,  there  may  also  be  alternative  tasks  that  warrant 
consideration.  The  several  mission  tasks  may  be  assigned  weight 
factors  in  accordance  with  an  appraisal  of  their  importance  and 
each  system  in  the  comparison  could  be  rated  according  to  its 
capability  in  each  of  the  mission  tasks.  The  index  of  merit 
for  each  system  would  then  be  given  by  an  equation  such  as  dis¬ 
cussed  in  the  Appendix. 

o  Future  Potential:  Although  the  several  systems  under  comparison 
can  all  perform  the  prescribed  missions,  they  may  differ  in 
their  potential  for  (a)  growth  in  performance,  and  (b)  unique 
performance  capability.  A  rating  system  such  as  discussed  in 
the  Appendix  may  again  be  utilized. 

o  Installation  and  Operational  Qualities:  Qualities  of  a  system, 
such  as  ease  of  installation,  maintenance,  operation,  and  favor¬ 
able  interface  with  other  systems,  should  be  recognized.  Again 
a  fig  -.re  of  merit  as  computed  from  an  equation  such  as  described 
in  the  Appendix  may  adequately  represent  this  item. 

o  Special  Comment:  This  space  is  only  filled  when  a  special  con¬ 
sideration  crucial  to  the  decision  is  pertinent. 
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MATRIX  REDUCTION  AND  JUDGMENT  PROCESS 

Table  I  illustrates  the  appearance  of  a  decision  matrix  of  the 
type  described.  It  involves  eight  rows  and  a  number  of  columns  repre¬ 
senting  each  of  the  competing  systems,  and  it  presents  a  formidable 
array  of  numbers  to  the  eye.  As  pointed  out  in  the  introduction,  an 
attempt  to  compress  this  information  into  a  single  number  for  each 
system  by  employing  a  figure  of  merit  scheme  (see  Appendix)  is  not 
satisfactory  because  (1)  it  involves  the  mixing  of  numbers  of  basically 
different  kinds  and  (2)  it  hides  some  of  the  information  needed  for  a 
proper  decision.  (By  the  same  token  it  must  be  borne  in  mind  that 
equal  rating  numbers  for  different  parameters  in  Table  1  do  not  carry 
equal  importance  and  that  the  relative  importance  of  these  parameters 
changes  with  change  in  mission.)  Instead  of  the  figure  of  merit  method 
for  reducing  the  size  of  the  matrix,  another  approach  is  presented  in 
this  paper. 
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Table  1 


ILLUSTRATIVE  DECISION  MATRIX 


Application  of  Propulsion  Systems  to  Mission  X 
(Mission  Initial  Operational  Date _ ) 


Parameters 


1.  System  R&D  Cost 

2.  Mission  Cost  Effectiveness 

3.  Decision  Date 

4.  Confidence  in  Meeting  IOC  Date 

5.  Flexibility 
0.  Potential 

7.  Installation  and  Operational 

Qualities 

8.  Special  Comment 


Systems 


It  •  1 


-If-  '  I 
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Table  2 

Rules  for  Matrix  Reduction 


1.  When  the  values  for  a  given  parameter  in  the  matrix  for 
the  various  systems  differ  by  an  amount  less  than  the  ac¬ 
curacy  of  the  estimate,  these  values  are  deleted  from  the 
matrix. 

2.  A  system  that  is  poorer  than  another  on  all  remaining  para¬ 
meters  has  all  its  numbers  deleted. 

3.  A  zero  for  any  quality  parameter  (e.g.,  items  4  to  7  of 
Table  1  inclusive)  deletes  the  system.  A  zero  is  assigned 
to  c  parameter  only  when  the  system  is  unacceptable  with 
respect  to  an  item  under  this  parameter  that  is  crucial. 

The  application  of  these  rules  will  be  illustrated  for  the  hy¬ 
pothetical  case  shown  in  Table  2,  The  parameters  associated  with 
five  competitive  systems  in  Mission  X  are  listed  as  part  of  an  exer¬ 
cise  to  determine  the  appropriate  developmental  decisions. 

The  reduced  matrix  is  shown  in  Table  3  The  mission  costs  and 
system  R&D  costs  were  eliminated  because  the  differences  were  con¬ 
sidered  to  be  less  than  the  accuracy  of  the  estimates.  System  E  was 
eliminated  because  of  the  zero  in  the  flexibility  parameter.  With 
column  E  removed,  the  remaining  values  in  rows  5  and  7  differ  by  less 
than  the  accuracy  of  the  estimate  and  therefore  were  also  eliminated. 
Systems  A  and  D  were  eliminated  because  the  remaining  values  were  less 
than  for  Systems  B  or  C.  We  now  have  left  only  two  systems,  B  and  C, 
the  first  with  an  advantage  in  confidence  of  u.  ‘ting  the  IOC  date  aud 
the  second  with  an  advantage  in  potential.  The  decision  dates  were 
left  in  the  matrix  to  facilitate  discussion. 

The  evaluation  of  the  data  in  Table  3  is  then  performed  with  the 
aid  of  the  following  rules. 
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Table  2 

ILLUSTRATIVE  COMPARISON  OF  SYSTEMS  IN  MISSION  X 
(Mission  IOC  date  1978) 


Parameters 

Systems 

A 

B 

C 

D 

E 

1. 

System  R&D  Cost  $10^ 

50 

55 

60 

56 

54 

2. 

Mission  Cost  $10^ 

1003 

1010 

1004 

1000 

1000 

3, 

Decision  Date 

1970 

1972 

1971 

1970 

1970 

4. 

Confidence  of  Meeting  IOC  Date 

.70 

.95 

.60 

.60 

.70 

5. 

Flexibility 

.7 

.7 

.8 

.7 

0 

Future  Potential 

.5 

.5 

.5 

.4 

7. 

Installation  and  Operational 
Qualities 

.6 

.6 

.7 

.7 

.8 

8. 

Special  Comment 

-10- 


Table  3 

REDUCED  MATRIX  CORRESPONDING  TO  TABLE  2 


ABC 


1.  System  R&D  Cost  $10^ 

2.  Mission  Cost  $10^ 

3.  Decision  Date 

4.  Confidence  of  Meeting  IOC  Date 

5.  Flexibility 

6.  Future  Potential 

7.  Installation  and  Operational 

Qualities 


1972  1971 

.95  .60 

.5  .9 


8.  Special  Comment 


Judgment  Guide  Rules 


A.  IOC  Date  Rules 

(1)  When  a  firm  IOC  date  exists  for  an  important  mission  applica¬ 
tion,  then  a  significant  advantage  in  ‘'confidence  to  meet  IOC  date" 
takes  precedence  over  modest  advantages  in  other  items. 

(2)  When  a  firm  IOC  date  exists  for  an  important  mission  applica¬ 
tion  and  the  two  contending  systems  ara,  (a)  a  conventional  system  with 
high  confidence  of  meeting  the  IOC  date  and,  (b)  a  novel  system  with 
much  greater  future  promise,  but  with  less  confidence  of  meeting  the 
IOC  date,  then  research  and  exploratory  engineering  should  be  performed 
on  the  novel  system,  at  least  up  to  the  date  of  development  decision, 
to  provide  an  improved  basis  for  a  judgment  of  confidence  in  meeting 
ths  IOC  date.  This  effort  should,  of  course,  continue  even  if  a  deci¬ 
sion  were  made  to  develop  the  conventional  system  for  the  specific  ap¬ 
plication  under  consideration  if,  at  the  date  of  decision,  the  results 
on  the  novel  system  are  encouraging  and  important  future  applications 
are  envisioned. 

(3)  When  a  firm  IOC  date  for  a  mission  application  does  not  exist, 
then  emphasis  shifts  to  qualities  like  "future  promise"  provided  that 
rating  disadvantages  in  other  items  are  not  excessive.  However,  the 
decision  is  limited  to  the  performance  of  research  and  exploratory 
engineering. 

(4)  When  the  confidence  of  meeting  the  IOC  date  is,  at  most, 
moderate  for  the  several  systems  under  comparison,  then  at  least  two 
systems  that  are  acceptable  on  other  items  should  be  supported  in  re¬ 
search  and  exploratory  engineering,  at  least  to  the  date  of  develop¬ 
ment  decision  and  possibly  beyond  this  date,  if  confidence  remains 
moderate. 

B.  Cost  Rules 


(1)  A  large  advantage  for  a  given  system  in  total  cost  for  a 
closely  specified  mission  may  be  significant  if  there  is  much  confidence 
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in  the  magnitude  of  the  mission  task  predicated  in  the  analysis  or  in 
the  manner  of  performing  the  mission.  However,  absence  of  this  confi¬ 
dence  may  shift  emphasis  to  other  comparison  items. 

(2)  When  the  system  R&D  cost  is  very  small  compared  with  the  mis¬ 
sion  cost,  then,  of  course,  a  substantial  advantage  in  mission  cost 
would  prevail  over  an  advantage  in  system  R&B  cost. 

(3)  When  the  system  R&D  cost  is  a  large  part  of  the  mission  cost 
and  when  the  magnitude  of  the  mission  task  is  uncertain  and  may  be 
small,  then  a  large  advantage  in  system  R&D  cost,  which  in  a  sense 
represents  the  initial  investment  risk,  should  strongly  influence  the 
decision. 

(4)  When  only  minor  differences  in  rating  on  all  parameters  occurs 
among  the  systems  under  consideration,  obviously  one  would  favor  the 
system  having  the  better  ratings  for  the  parameters  of  major  importance 
to  the  mission  under  consideration.  It  is  harder  in  this  case  to  iden¬ 
tify  the  optimum  system  with  certainty,  but,  by  the  same  token,  the 
consequence  of  not  choosing  the  optimum  system  is  not  serious. 

The  example  shown  in  Tables  2  and  3  comes  under  the  purview  of 
Rule  A-2.  While  System  B  shows  a  large  advantage  in  confidence  of 
meeting  the  IOC  date.  System  C  promises  considerable  improvement  in 
future  potential.  Thus,  if  there  is  much  confidence  in  the  future 
utility  of  the  proposed  Mission  X,  then  the  indicated  decisions  are: 

(a)  Do  research  and  exploratory  engineering  on  System  C  until  at 
least  1971,  at  which  time  a  reappraisal  of  the  confidence  of 
meeting  the  IOC  date  is  made  for  this  system. 

(b)  If  the  confidence  for  System  C  is  still  low,  a  decision  is 
appropriate  for  developing  System  B  for  Mission  X.  A  decision 
to  continue  or  stop  the  effort  on  System  C  would  depend  on  the 
amount  of  progress  made  and  the  importance  of  its  future  appli¬ 
cation  prospects. 

(c)  If,  on  the  other  hand,  sufficient  progress  is  made  on  System  C 
by  1971  to  indicate  a  confidence  of  meeting  the  required  IOC 
date  comparable  to  that  of  Mission  B  and  a  re-evaluation  of 
the  costs  on  the  basis  of  this  new  information  still  shows  no 
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no  critical  cost  differences,  then  one  would  choose  System 
C  for  development  for  Mission  X. 

The  rules  for  matrix  reduction  presented  in  this  paper  represent 
in  my  mind  only  a  beginning.  They  will  not  handle  all  cases.  Hence, 
amplification  and  refinement  of  the  rules  can  be  anticipated  as  exper¬ 
ience  with  the  process  grows.  The  rules  for  the  judgment  process  are 
based  on  about  35  years  of  personal  experience  in  research  and  develop¬ 
ment  on  flight  propulsion  systems  and  the  emphasis  in  the  rules  pre¬ 
sented  herein  reflects  this  somewhat  narrow  field  of  decision  practice. 
(Some  of  the  background  for  these  rules  appears  in  Ref.  2).  The  rules 
are  admittedly  arbitrary  and  limited  because  they  represent  a  single 
observer's  viewpoint  in  a  very  complex  field.  Again  one  must  consider 
these  rules  as  representing  only  a  beginning  to  be  amplified  and  modi¬ 
fied  as  expertise  and  range  of  application  grow. 


y 
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Appendix 
Figures  of  Merit 

It  is  common  practice  to  use  the  following  equation  for  establish 
ing  a  figure  of  merit  in  the  comparison  of  various  systems. 


j  r n  j  i  n 

i  -  £  R.  W  /  E  W 

Li-i  \J  i-i  1 


(i) 


where 


rating  index  of  system, 

rating  of  J*"*1  system  on  the  iC^  comparison  para¬ 


meter 


W. 

x 


.th 


weighting  factor  for  the  i  comparison  parameter 
•number  of  comparison  parameters 


If  the  rating  R*?  is  assigned  a  value  between  zero  and  1  where  1 

*  j 

represents  a  rating  of  excellent,  then  I  is  a  number  between  0  and  1. 

An  objection  to  Eq.  (1)  is  that  when  a  large  number  of  rating  parameters 
are  involved,  a  zero  rating  on  a  crucial  parameter,  which  should  have 
eliminated  the  system,  would  have  only  a  minor  effect  on  the  overall 
rating  value.  This  problem  can  be  eliminated  by  employing  the  follow¬ 
ing  equation 


Li  '  <) 


n 

I 

i-1 


W, 


(2) 


where  the  value  of  R?  lies  between  c  and  1.  The  value  of  c  is  assumed 

1  * 

to  approach  zero  as  a  limit  and  is  assigned  to  a  parameter  only  when  the 
system  under  discussion  is  unacceptable  on  this  parameter.  Equation 
(2)  has  the  objection  (which  is  shared  also  by  Eq.  (1))  that  it  re¬ 
quires  modification  if  relatedness  exists  between  any  of  the  parameters. 

This  refinement  is  introduced  to  avoid  the  problem  of  dividing  by 
zero  in  Eq.  (2)  which  mathematical  purists  say  is  not  a  defined  operation. 
A  value  of  f  (or  0  in  the  case  of  Eq.  (1))  is  assigned  when  the  system  is 
unacceptable  on  a  crucial  parameter. 
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